1. Articles
  2. GOV.UK amendments

Amendments to GOV.UK statistics publications

An exploratory data analysis to inform Reproducible Analytical Pipelines, by Duncan Garmonsway on 15 January 2019.

Introduction

The Reproducible Analytical Pipeline (RAP) is an alternative production methodology for automating the bulk of steps involved in creating a statistical report.

One motivation to use RAP methods is to avoid making trivial mistakes, by removing as much opportunity for human error as possible. For example, the following note describes a trivial amendment that could have been avoided by programming a computer to update the date of the quarterly publication.

The date for the publication … was incorrectly stated as 14 February 2019, this has now been corrected to 21 February 2019 which is in line with the release calendar and established practice to publish on the third Thursday of the month.

This article investigates.

Detection of amendments

Updates are regarded as ‘amendments’ when they contain a word from a list. The list of words was compiled in discussion with colleagues and by referring to a thesaurus. Some words were discarded because there were many false positives.

The number of notes that matched each word in the vocabulary

## # A tibble: 31 x 2
##    pattern               n
##    <chr>             <int>
##  1 correct             395
##  2 amend               385
##  3 error               189
##  4 erratum              16
##  5 fix                  11
##  6 mistake               4
##  7 omission              4
##  8 wrong                 4
##  9 inadvertent           3
## 10 inaccuracy            1
## 11 misinterpretation     1
## 12 rectify               1
## 13 aberration            0
## 14 blunder               0
## 15 deviation             0
## 16 falsehood             0
## 17 flaw                  0
## 18 glitch                0
## 19 lapse                 0
## 20 misapplication        0
## 21 misapprehension       0
## 22 miscalculation        0
## 23 misconception         0
## 24 misjudgment           0
## 25 misprint              0
## 26 misstatement          0
## 27 misstep               0
## 28 misunderstanding      0
## 29 overestimation        0
## 30 oversight             0
## 31 remedy                0

Searchable table of all notes

How common are changes (not just amendments)?

Many first publications were backdated. The date of the first change is a reasonable estimate for the date that GOV.UK first published statistics

Changes per month

Changes per year

How common are amendments?

The most that can be said is that “amendments happen”. The number of amendments detected is in fact quite low (about 2% of all changes), and that could be because:

On the other hand, RAP won’t necessarily reduce the number of amendments – in fact it might increase it by having greater power to detect mistakes through peer review. Errors are likely to be noticed when RAP is first applied.

Amendments per month

Amendments per year

Amendments as percentage of all changes per month

Amendments as percentage of all changes per year

Distribution of rates per organisation in the last complete year

The graph shows for each organisation:

Most organisations have made no amendments. Some of those haven’t published much, but a few have published a lot. Few organisations have a credible interval entirely above a 5% amendment rate.

The Empirical Bayes model was fitted with a beta-binomial prior fitted to the data by maximum likelihood estimation; see ?ebbr::ebb_fit_prior for details.

Why are amendments made?

What other changes are made?

Session info

## ─ Session info ─────────────────────────────────────────────────────────────────────────
##  setting  value                       
##  version  R version 3.6.1 (2017-01-27)
##  os       Ubuntu 16.04.6 LTS          
##  system   x86_64, linux-gnu           
##  ui       X11                         
##  language en_US.UTF-8                 
##  collate  en_US.UTF-8                 
##  ctype    en_US.UTF-8                 
##  tz       UTC                         
##  date     2019-08-13                  
## 
## ─ Packages ─────────────────────────────────────────────────────────────────────────────
##  package     * version    date       lib source                                   
##  assertthat    0.2.1      2019-03-21 [1] CRAN (R 3.6.1)                           
##  backports     1.1.4      2019-04-10 [1] CRAN (R 3.6.1)                           
##  broom         0.5.2      2019-04-07 [1] CRAN (R 3.6.1)                           
##  callr         3.3.1      2019-07-18 [1] CRAN (R 3.6.1)                           
##  cellranger    1.1.0      2016-07-27 [1] CRAN (R 3.6.1)                           
##  cli           1.1.0      2019-03-19 [1] CRAN (R 3.6.1)                           
##  codetools     0.2-16     2018-12-24 [3] CRAN (R 3.6.1)                           
##  colorspace    1.4-1      2019-03-18 [1] CRAN (R 3.6.1)                           
##  crayon        1.3.4      2017-09-16 [1] CRAN (R 3.6.1)                           
##  crosstalk     1.0.0      2016-12-21 [1] CRAN (R 3.6.1)                           
##  desc          1.2.0      2018-05-01 [1] CRAN (R 3.6.1)                           
##  devtools      2.1.0      2019-07-06 [1] CRAN (R 3.6.1)                           
##  digest        0.6.20     2019-07-04 [1] CRAN (R 3.6.1)                           
##  dplyr       * 0.8.3      2019-07-04 [1] CRAN (R 3.6.1)                           
##  DT          * 0.8        2019-08-07 [1] CRAN (R 3.6.1)                           
##  ebbr        * 0.1        2019-08-13 [1] Github (dgrtwo/ebbr@4b9747d)             
##  ellipsis      0.2.0.1    2019-07-02 [1] CRAN (R 3.6.1)                           
##  evaluate      0.14       2019-05-28 [1] CRAN (R 3.6.1)                           
##  fansi         0.4.0      2018-10-05 [1] CRAN (R 3.6.1)                           
##  forcats     * 0.4.0      2019-02-17 [1] CRAN (R 3.6.1)                           
##  fs            1.3.1      2019-05-06 [1] CRAN (R 3.6.1)                           
##  generics      0.0.2      2018-11-29 [1] CRAN (R 3.6.1)                           
##  ggplot2     * 3.2.1      2019-08-10 [1] CRAN (R 3.6.1)                           
##  glue          1.3.1      2019-03-12 [1] CRAN (R 3.6.1)                           
##  govdown       0.5.0.9000 2019-08-13 [1] Github (ukgovdatascience/govdown@f7f5bd7)
##  gtable        0.3.0      2019-03-25 [1] CRAN (R 3.6.1)                           
##  haven         2.1.1      2019-07-04 [1] CRAN (R 3.6.1)                           
##  here        * 0.1        2017-05-28 [1] CRAN (R 3.6.1)                           
##  hms           0.5.0      2019-07-09 [1] CRAN (R 3.6.1)                           
##  htmltools     0.3.6      2017-04-28 [1] CRAN (R 3.6.1)                           
##  htmlwidgets   1.3        2018-09-30 [1] CRAN (R 3.6.1)                           
##  httpuv        1.5.1      2019-04-05 [1] CRAN (R 3.6.1)                           
##  httr          1.4.1      2019-08-05 [1] CRAN (R 3.6.1)                           
##  jsonlite      1.6        2018-12-07 [1] CRAN (R 3.6.1)                           
##  knitr         1.24       2019-08-08 [1] CRAN (R 3.6.1)                           
##  labeling      0.3        2014-08-23 [1] CRAN (R 3.6.1)                           
##  later         0.8.0      2019-02-11 [1] CRAN (R 3.6.1)                           
##  lattice       0.20-38    2018-11-04 [3] CRAN (R 3.6.1)                           
##  lazyeval      0.2.2      2019-03-15 [1] CRAN (R 3.6.1)                           
##  lubridate   * 1.7.4      2018-04-11 [1] CRAN (R 3.6.1)                           
##  magrittr      1.5        2014-11-22 [1] CRAN (R 3.6.1)                           
##  memoise       1.1.0      2017-04-21 [1] CRAN (R 3.6.1)                           
##  mime          0.7        2019-06-11 [1] CRAN (R 3.6.1)                           
##  modelr        0.1.5      2019-08-08 [1] CRAN (R 3.6.1)                           
##  munsell       0.5.0      2018-06-12 [1] CRAN (R 3.6.1)                           
##  nlme          3.1-140    2019-05-12 [3] CRAN (R 3.6.1)                           
##  patchwork   * 0.0.1      2019-08-13 [1] Github (thomasp85/patchwork@fd7958b)     
##  pillar        1.4.2      2019-06-29 [1] CRAN (R 3.6.1)                           
##  pkgbuild      1.0.4      2019-08-05 [1] CRAN (R 3.6.1)                           
##  pkgconfig     2.0.2      2018-08-16 [1] CRAN (R 3.6.1)                           
##  pkgload       1.0.2      2018-10-29 [1] CRAN (R 3.6.1)                           
##  prettyunits   1.0.2      2015-07-13 [1] CRAN (R 3.6.1)                           
##  processx      3.4.1      2019-07-18 [1] CRAN (R 3.6.1)                           
##  promises      1.0.1      2018-04-13 [1] CRAN (R 3.6.1)                           
##  ps            1.3.0      2018-12-21 [1] CRAN (R 3.6.1)                           
##  purrr       * 0.3.2      2019-03-15 [1] CRAN (R 3.6.1)                           
##  R6            2.4.0      2019-02-14 [1] CRAN (R 3.6.1)                           
##  Rcpp          1.0.2      2019-07-25 [1] CRAN (R 3.6.1)                           
##  readr       * 1.3.1      2018-12-21 [1] CRAN (R 3.6.1)                           
##  readxl        1.3.1      2019-03-13 [1] CRAN (R 3.6.1)                           
##  remotes       2.1.0      2019-06-24 [1] CRAN (R 3.6.1)                           
##  rlang         0.4.0      2019-06-25 [1] CRAN (R 3.6.1)                           
##  rmarkdown     1.14       2019-07-12 [1] CRAN (R 3.6.1)                           
##  rprojroot     1.3-2      2018-01-03 [1] CRAN (R 3.6.1)                           
##  rstudioapi    0.10       2019-03-19 [1] CRAN (R 3.6.1)                           
##  rvest         0.3.4      2019-05-15 [1] CRAN (R 3.6.1)                           
##  scales        1.0.0      2018-08-09 [1] CRAN (R 3.6.1)                           
##  sessioninfo   1.1.1      2018-11-05 [1] CRAN (R 3.6.1)                           
##  shiny         1.3.2      2019-04-22 [1] CRAN (R 3.6.1)                           
##  stringi       1.4.3      2019-03-12 [1] CRAN (R 3.6.1)                           
##  stringr     * 1.4.0      2019-02-10 [1] CRAN (R 3.6.1)                           
##  testthat      2.2.1      2019-07-25 [1] CRAN (R 3.6.1)                           
##  tibble      * 2.1.3      2019-06-06 [1] CRAN (R 3.6.1)                           
##  tidyr       * 0.8.3      2019-03-01 [1] CRAN (R 3.6.1)                           
##  tidyselect    0.2.5      2018-10-11 [1] CRAN (R 3.6.1)                           
##  tidyverse   * 1.2.1      2017-11-14 [1] CRAN (R 3.6.1)                           
##  usethis       1.5.1      2019-07-04 [1] CRAN (R 3.6.1)                           
##  utf8          1.1.4      2018-05-24 [1] CRAN (R 3.6.1)                           
##  vctrs         0.2.0      2019-07-05 [1] CRAN (R 3.6.1)                           
##  VGAM          1.1-1      2019-02-18 [1] CRAN (R 3.6.1)                           
##  withr         2.1.2      2018-03-15 [1] CRAN (R 3.6.1)                           
##  xfun          0.8        2019-06-25 [1] CRAN (R 3.6.1)                           
##  xml2          1.2.2      2019-08-09 [1] CRAN (R 3.6.1)                           
##  xtable        1.8-4      2019-04-21 [1] CRAN (R 3.6.1)                           
##  yaml          2.2.0      2018-07-25 [1] CRAN (R 3.6.1)                           
##  zeallot       0.1.0      2018-01-28 [1] CRAN (R 3.6.1)                           
## 
## [1] /home/travis/R/Library
## [2] /usr/local/lib/R/site-library
## [3] /home/travis/R-bin/lib/R/library